Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #23851: first auto-bond implementation #439

Merged
merged 1 commit into from Jul 10, 2018

Conversation

Thomas-Gelf
Copy link
Contributor

refs #23851

@theforeman-bot
Copy link
Member

Do not merge! This patch has not been tested yet.

Can an existing organization member please verify this patch?

@theforeman-bot
Copy link
Member

Issues: #23851

Copy link
Member

@timogoebel timogoebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Thomas-Gelf: Thanks, looks good. Do you mind adding a couple of tests for this?

@@ -66,6 +67,10 @@ def self.discovery_hostname_fact_array
from_array Setting['discovery_hostname']
end

def self.discovery_auto_bond?
Foreman::Cast.to_bool(Setting['discovery_auto_bond'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? I believe the value should already be a boolean.
In addition, this is not used anywhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, just added it as it's being done the same way the line below 😆

@@ -1,9 +1,12 @@
require 'foreman_discovery/lldp_neighbors'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be necessary. Rails does auto loading pretty well. Let's remove this.
You might want to move the library to app/services as well (instead of lib/foreman_discovery/).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It drove me mad to figure out what to place where, as I faced quite some inconsistencies. Well, at least I haven't been able to really grasp the rules. No wonder Ruby projects fire an such an immense amount of stat-calls.

I placed this code in a separate class to make it easier to test in an isolated way. Just, I failed to run the tests, gave some rake test:foreman_discovery examples triggered from the foreman base directory a try, but nothing happened.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timogoebel: any helpful hint on how to kick the test run would be greatly appreciated!

neighbors.set name, properties
end

return neighbors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: superfluous return

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

)

bond.save!
host.interfaces.push bond
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This needs brackets...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call to push? Do you have a brackets-everywhere policy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(because then there would be more)

list.push name if neighbor['PVID'] == interface['PVID']
end

list.size > 1 ? list : nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:
list.empty? ? nil : list

or

return if list.empty?
list

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure... I had the second one first, but I hate it to read code where it's not immediately obvious that nil/null must be expected. And what's ugly about this is that it's not list.empty? - only results with more than one entry are returned, if there is only one peer it should return nil.

I changed it to read:

      return if list.size < 2
      list

This way it's more obvious that at least two are required for a valid candidate.

Cheers,
Thomas

@Thomas-Gelf
Copy link
Contributor Author

LldpNeighbors is covered, HostConverter will follow

@Thomas-Gelf
Copy link
Contributor Author

NB: I tried to address all of the above, just the question related to brackets still stands

@Thomas-Gelf
Copy link
Contributor Author

# Running:

LldpNeighborsTest#test_0001_#get_neighbors_by_interface gives nothing with no LLDP facts = 0.77 s = .
LldpNeighborsTest#test_0003_#get_neighbors_by_interface gives a valid pair for LLDP facts with matching PVID = 0.00 s = .
LldpNeighborsTest#test_0006_#list_by_pvid pairs interfaces by PVID = 0.00 s = .
LldpNeighborsTest#test_0007_#list_by_pvid supports multiple PVIDs = 0.00 s = .
LldpNeighborsTest#test_0002_#get_neighbors_by_interface gives nothing with unmatching PVID in LLDP facts = 0.00 s = .
LldpNeighborsTest#test_0005_#list_by_pvid gives an empty Hash without LLDP facts = 0.00 s = .
LldpNeighborsTest#test_0004_#list_by_pvid gives an empty Hash without LLDP facts = 0.00 s = .

Finished in 0.783887s, 8.9299 runs/s, 12.7569 assertions/s.
7 runs, 10 assertions, 0 failures, 0 errors, 0 skips

@timogoebel
Copy link
Member

just the question related to brackets still stands

We don't have a policy for that, but I'd prefer brackets everywhere except when the method is called without any arguments.

end

test "#list_by_pvid gives an empty Hash without LLDP facts" do
assert_equal Hash({}), simple_facts.list_by_pvid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the only case were i'd call assert_equal with brackets. Please omit the Hash() everywhere, it's not good ruby style.

end

def lldp_facts
neighbors'facts_with_lldp'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing whitespace

test "#list_by_pvid supports multiple PVIDs" do
assert_equal Hash('182' => %w(eth2), '184' => %w(eth1)), lldp_facts.list_by_pvid
end

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the methods below can be private, a better (or modern) way to write this would be using let. But this is fine as well.

context 'with facts' do
 let(:facts) { some code } # acts as a lazy accessor
  test 'abc' do
    assert facts
  end
end

@Thomas-Gelf Thomas-Gelf force-pushed the feature/auto-bond branch 2 times, most recently from c7168ab to f3911c4 Compare June 8, 2018 09:35
@Thomas-Gelf
Copy link
Contributor Author

Also extended host_discovery_test:

HostDiscoveredTest#test_0033_provision_interface isn't touched with no LLDP facts = 0.87 s = .
HostDiscoveredTest#test_0034_provision_interface isn't touched with no peer on the same VLAN = 0.77 s = .
HostDiscoveredTest#test_0036_provision_interface is not switched to bond0 if disabled = 0.77 s = .
HostDiscoveredTest#test_0035_provision_interface is switched to bond0 with more than one interface on the same VLAN = 1.00 s = .

@Thomas-Gelf
Copy link
Contributor Author

There is one more:

HostDiscoveredTest#test_0037_former provision_interface is not managed after switching to bond0 = 0.95 s = .

@Thomas-Gelf Thomas-Gelf force-pushed the feature/auto-bond branch 2 times, most recently from bc4ded0 to d208bd6 Compare June 8, 2018 09:57
@Thomas-Gelf Thomas-Gelf changed the title WIP: first auto-bond implementation Fixes #23851: first auto-bond implementation Jun 8, 2018
@Thomas-Gelf
Copy link
Contributor Author

@timogoebel: tests look good to me, please let me know in case you need more

@timogoebel
Copy link
Member

ok to test

Copy link
Member

@timogoebel timogoebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Thomas-Gelf
Copy link
Contributor Author

Don't blame ReferenceError: Can't find variable: jQuery (Capybara::Poltergeist::JavascriptError) on me ;-)

@timogoebel
Copy link
Member

I checked this out. Making the changes in memory is actually pretty easy.

My naive approach looks like this:

 diff --git a/app/services/foreman_discovery/host_converter.rb b/app/services/foreman_discovery/host_converter.rb
index bcb1c05..25fa91e 100644
--- a/app/services/foreman_discovery/host_converter.rb
+++ b/app/services/foreman_discovery/host_converter.rb
@@ -33,26 +33,19 @@ class ForemanDiscovery::HostConverter

     ip = primary.ip
     name = primary.name
-    primary.update(
-       :primary   => false,
-       :provision => false,
-       :managed   => false,
-       :name      => nil,
-       :ip        => nil
-    )
+    primary.name = nil
+    primary.ip = nil
+    primary.provision = false
+    primary.primary = false

-    bond = Nic::Bond.create(
-        :identifier       => "bond0",
-        :attached_devices => neighbors,
-        :primary          => true,
-        :provision        => true,
-        :name             => name,
-        :ip               => ip,
-        :host             => host
+    bond = host.interfaces.build(
+      type: 'Nic::Bond',
+      identifier: 'bond0',
+      primary: true,
+      provision: true,
+      name: name,
+      ip: ip
     )
-
-    bond.save!
-    host.interfaces.push(bond)
   end

   def self.set_build_clean_facts(host)

This issue now is, that the primary interface cannot be found as rails thinks the relation is nil, see https://github.com/theforeman/foreman/blob/fcc2b55d07cbb3708f2fab212bf57a3366561d06/app/models/host/base.rb#L22
I have no idea yet, how we can workaround this.

@timogoebel
Copy link
Member

@Thomas-Gelf: I like adding the idea of adding this to the fact parser so this code runs when the host is saved to the database. But I don't think it makes a lot of sense in core as the fact parser in core should just provide the data from a system so it can be displayed in Foreman's UI. In discovery we need this before provisioning happens. The fact parser prepares the host for proper provisioning.
We already have our own fact parser where we should be able to override the interfaces method to get this working.

@lzap, what do you think?

@lzap
Copy link
Member

lzap commented Jun 20, 2018

Overriding our parser is the best idea IMHO so far. :)

@Thomas-Gelf
Copy link
Contributor Author

Update: I gave the fact_parser a try, shifted the whole logic - but failed. Have been unable to enforce a Nic::Bond and to pass attached_devices. Eventually I did it wrong, but anyways: I moved the call to eventually_make_bond() to Host::Discovered.populate_discovery_fields_from_facts. This way the transformation takes place immediately once the interfaces are being stored for the first time.

@Thomas-Gelf
Copy link
Contributor Author

Hint: my attempt with the fact_parser was solely in foreman_discover, I didn't tweak the base class in foreman. So, by enhancing the possibilities there this should also be possible. However, the new approach looks good to me. @lzap: thoughts?

@Thomas-Gelf Thomas-Gelf force-pushed the feature/auto-bond branch 2 times, most recently from 71aaa6e to 2a577f1 Compare June 22, 2018 15:00
@Thomas-Gelf
Copy link
Contributor Author

Rebased and fixed mac address setting

Copy link
Member

@lzap lzap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok looks better but I want you to move the eventually_make_bond method somewhere else please.

@@ -143,6 +143,7 @@ def populate_discovery_fields_from_facts(facts)
self.discovery_attribute_set = DiscoveryAttributeSet.where(:host_id => id).first_or_create
self.discovery_attribute_set.update_attributes(import_from_facts)
# set additional discovery attributes
::ForemanDiscovery::HostConverter.eventually_make_bond(self) if Setting[:discovery_auto_bond]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This place is better but the code does not belong into HostConverter.

@@ -19,6 +19,43 @@ def self.to_managed(original_host, set_managed = true, set_build = true, added_a
host
end

def self.eventually_make_bond(host)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this method out of HostConverter into LldpNeighbors or somewhere else? I suggest creating a "post processing framework":

  • ForemanDiscovery::ImportHookService - a simple registry of hooks
  • ForemanDiscovery::ImportHook - base class for processor - keep it simple after_populate(host, facts) as the only method for now
  • ForemanDiscovery::ImportHooks::LldpNeighbor - implementation - move this method there

Optionally break method populate_discovery_fields_from_facts into three other separate hooks: DiscoveryAttribute, SubnetAndTaxonomy and LockTemplates. I can do this later if you don't want to touch this.

The reason I want this is in the future we might break the processing of NICs and Facts in Core in similar way, so we could easily merge this. The idea is not to save anything into database during the whole import phase so plugins can hook into the process easily and at the very end of the whole processing, save all NICs and fields to database in a single place.

@Thomas-Gelf
Copy link
Contributor Author

@lzap: I moved the code as requested and adjusted tests accordingly. Regarding the "post processing framework": could you please point me to an example Hook in the Foreman code, just to get an idea of how that should look like? In case it is not tricky I could give it a try.

@timogoebel
Copy link
Member

I'd suggest to implement the "post processing framework" in a separate PR. It's not related to this issue. I agree it's a nice refactoring, but we don't need it now. @lzap, agreed?

Copy link
Member

@lzap lzap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, although I think better direction would be creating post-processing framework. Please document the changes in https://github.com/theforeman/theforeman.org/blob/gh-pages/plugins/foreman_discovery/nightly/index.md and we are good to merge.

@timogoebel
Copy link
Member

Ok, although I think better direction would be creating post-processing framework.

I agree, luckily we can still do that later.

Please document the changes

Here you go: theforeman/theforeman.org#1129

@lzap lzap merged commit 82809b7 into theforeman:develop Jul 10, 2018
@lzap
Copy link
Member

lzap commented Jul 10, 2018

Sure, I'd rather go that direction sooner because discovery plugin is basically a collection of hacks :-)

@Thomas-Gelf Thomas-Gelf deleted the feature/auto-bond branch July 10, 2018 08:32
@timogoebel
Copy link
Member

Thanks, @Thomas-Gelf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants